Skip to content

Add LMFDB concepts#8

Merged
katjabercic merged 6 commits into
mainfrom
feat/lmfdb-slurper
May 7, 2026
Merged

Add LMFDB concepts#8
katjabercic merged 6 commits into
mainfrom
feat/lmfdb-slurper

Conversation

@katjabercic
Copy link
Copy Markdown
Owner

Adds the L-functions and Modular Forms Database (LMFDB) as a new source.

  • New slurper.source_lmfdb.LmfdbSlurper reads kwl_knowls rows from the
    LMFDB Postgres connection (via the lmf package) and saves each knowl as
    an Item under the new LMFDB source, with https://www.lmfdb.org/knowledge/show/<id>
    as the URL.
  • New management commands import_lmfdb and clear_lmfdb.
  • New SlurperRun model + migration that records the last successful run
    per source. The LMFDB slurper is throttled to once every 7 days by default;
    both import_lmfdb and clear_lmfdb accept --force to override.
  • rebuild_db now invokes the LMFDB clear/import steps with messages
    explaining the throttle.
  • Item.identifier, name, and url raised from 200 → 300 chars (some
    Wikipedia entries were getting truncated); migration 0021 included.

@katjabercic katjabercic requested a review from Stanoja May 2, 2026 06:57
Stanoja added 2 commits May 6, 2026 21:18
-added requirement for lmfdb-dblite
Copy link
Copy Markdown
Collaborator

@Stanoja Stanoja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested and pushed 2 fixes:

  1. Reformatting
  2. Dependency for the lmf import

Everything else looks good

self.source = Item.Source.LMFDB

def fetch_rows(self):
from lmf import db
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an import for this, seems that the correct library is imported: https://github.com/roed314/lmfdb-lite

Comment thread web/requirements.txt
scispacy~=0.6.2
python-decouple~=3.8
unidecode~=1.4.0
lmfdb-lite[pgbinary] @ git+https://github.com/roed314/lmfdb-lite.git
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️ ...this dependency

@Stanoja Stanoja self-requested a review May 6, 2026 19:37
@katjabercic katjabercic merged commit 2098426 into main May 7, 2026
1 check passed
@katjabercic
Copy link
Copy Markdown
Owner Author

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants